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(54) Color image sensing apparatus and method of expanding dynamic range 



(57) Disclosed is an image processing method of 
expanding the dynamic range by determining a saturat- 
ed or noise region in an image signal and replacing the 
image signal with a proper image signal. A threshold val- 
ue (step S4) is set in units of colors on the basis of a 
standard image signal for each color, which is obtained 
upon sensing an image in a standard exposure, and a 
non-standard image signal for each color, which is ob- 
tained upon sensing the image in a non-standard expo- 
sure. A saturated or noise region is determined by ap- 
plying the threshold value set in units of colors to the 
standard image signal. A region determined as a satu- 
rated or noise region is replaced with the non-standard 
image signal. 
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Description 

BACKGROUND OF THE INVENTION 

The present invention relates to an image process- 
ing method and apparatus for processing an image ob- 
tained using, e.g., an image sensing device, and an im- 
age sensing apparatus. 

In recent years, image sensing devices such as 
CCD image sensing devices are popularly used as im- 
age reading elements for television cameras, video 
cameras, electronic cameras, and the like. However, the 
image sensing device is inferior to silver halide media 
in terms of the dynamic range, and an image obtained 
by an image sensing device has considerably low qual- 
ity depending on the photographing conditions. 

In order to expand the dynamic range of an image 
sensing device, a technique for sensing an image of an 
identical scene in different exposure amounts to obtain 
a plurality of image signals, and combining the plurality 
of image data by a predetermined calculation to obtain 
an image with an expanded dynamic range is proposed 
in, e.g., Japanese Patent Laid-Open No. 63-306778. 

Figs. 1 A to 1 C and Figs. 2A to 2C are graphs show- 
ing typical conventional methods of expanding the dy- 
namic range. 

The graph in Fig. 1A shows the relationship be- 
tween an illuminance (abscissa) and an output from the 
image sensing device (ordinate) obtained upon sensing 
a certain object. In Fig. 1 A, a characteristic I represents 
the relationship between an illuminance and an output 
from the device for an image obtained upon sensing the 
object in a proper exposure (to be referred to as a 
"standard image" hereinafter). A characteristic II repre- 
sents the relationship between an illuminance and an 
output from the device for an image obtained upon sens- 
ing the object in an over exposure (to be referred to as 
a "non-standard image" hereinafter). Referring to Fig. 
1A, signals below an output level n are regarded as 
noise. A characteristic I 1 is obtained when an output val- 
ue from the image sensing device (Y-axis) of the stand- 
ard image I is multiplied by a luminance level adjustment 
value K. 

A concept of image combining according to this 
method will be described. In Fig. 1 A, at a position "x^ 1 
on the axis representing the object illuminance, as for 
the image signal (image signal I) obtained in a proper 
exposure, the output from the image sensing device is 
at noise level. However, the image signal (image signal 
II) obtained in an over exposure has a proper value (ex- 
ceeds the noise level n). That is, when an output value 
between level 0 and the noise level n is replaced with 
the output value of the non-standard image ll : an image 
signal with an expanded dynamic range can be ob- 
tained. As shown in Fig. 1 A, the gradient of the standard 
image I obtained in a proper exposure is different from 
that of the non-standard image obtained in an over ex- 
posure. Therefore, when the signal value I is only re- 



placed with the signal value II, a discontinuous region is 
generated. For replacement, not the image signal I but 
an image signal I* is used. 

In Fig. 1 A, the image signal I' (to be also referred to 
5 as a standard image hereinafter) is obtained by multi- 
plying the standard image I by the predetermined lumi- 
nance level adjustment value K. The level adjustment 
value K is set such that the gradient of the standard im- 
age I' coincides with that of the non-standard image sig- 
io nal II. Processing of generating the standard image I' 
will be referred to as "luminance level adjustment" here- 
inafter. With this luminance level adjustment, a noise 
level n' of the standard image I' becomes n x K. 

An appropriate threshold value T is set above the 
*5 noise level n' n x K). As shown in Fig. 1B, data in a 
region below the threshold value T is replaced with the 
data of the non-standard image II. As for a region above 
the threshold value T; the data of the standard image I' 
is used without being processed. With this processing, 
20 noise in the section (n' - n) of the standard image I' is 
removed. The gradient of a combined image III (Fig. 1B) 
obtained in this manner corresponds to the gradient of 
the standard image I multiplied by K. Therefore, as 
shown in Fig. 1C, when the gradient of the combined 
25 image III is multiplied by 1/K to obtain the original gra- 
dient, the luminance level corresponding to the standard 
image I can be obtained. With the above processing, 
combined image data III' with an expanded dynamic 
range, i.e., at a low noise level can be obtained. 
30 The conventional technique for expanding the dy- 
namic range by decreasing noise regions has been de- 
scribed above. A technique for expanding the dynamic 
range by decreasing saturated regions will be described 
below with reference to Figs. 2A to 2C. 
35 Figs. 2A to 2C are graphs for explaining a technique 
for combining an image sensed in a proper exposure (to 
be referred to as a standard image IV hereinafter) with 
an image sensed in an under exposure (to be referred 
to as a standard image V hereinafter). 
40 When image data is represented as 8-bit data, and 
the output level exceeds 1 ,023, the image data is satu- 
rated. Referring to Fig. 2A, at "x2" on the axis represent- 
ing the object illuminance, the image data of the stand- 
ard image IV is saturated while the image data of the 
^5 standard image V has a proper value. Therefore, as 
shown in Fig. 2A, the gradient of a non-standard image 
V, which is obtained by multiplying the gradient of the 
standard image V by the luminance level adjustment 
value K\ i.e., by multiplying the data of the standard im- 
50 age V by K\ coincides with that of the standard image 
IV. As a result, the non-standard image V obtains a lu- 
minance level corresponding to that of the standard im- 
age IV. As in noise removal (Fig. 1A), an appropriate 
threshold value T' is set. The image data of the standard 
55 image IV in a region above the threshold value V is re- 
placed with the non-standard image V. As for a region 
below the threshold value T\ the image data of the 
standard image IV is used without being processed. 
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With this processing, the data of a combined image VI 
(Fig. 2B) has the same luminance level (gradient) as that 
of the standard image IV in Fig. 2A, though it has a 1 0-bit 
width. Hence, an appropriate knee point is set, as shown 
in Fig. 2C, and a bright region is compressed while main- 
taining the same gradient as that of the standard image 
IV 

With the above processing, image data VI' with an 
expanded dynamic range without saturation in a bright 
portion can be combined. 

In the above conventional techniques, a dark noise 
region or a saturated region of a standard image is de- 
termined using a predetermined threshold value (T or 
T'). In the conventional techniques, when the image 
sensing device has a color filter, an output signal from 
the image sensing device is used without being proc- 
essed to set a threshold value. 

However, output values from image sensing devic- 
es vary in accordance with colors because of the spec- 
tral sensitivity, so the dark noise or saturated region can- 
not be completely determined. This problem will be de- 
scribed below referring to a case wherein a saturated 
region of a standard image sensed in a proper exposure 
is determined and replaced with a corresponding region 
of a non-standard region. 

When the image sensing device has three color fil- 
ters for R, G, and B, saturated regions in each of R, G, 
and B image data may not be determined and separated 
unless the characteristics of each color filter is taken into 
consideration. For this reason, saturated regions unde- 
sirably remain in combined image data. In addition, it 
becomes difficult to set a threshold value because of 
variations in each color image signal. 

SUMMARY OF THE INVENTION 

The present invention has been made to solve the 
above problem, and has as a concern to provide an im- 
age processing method and apparatus, which can ap- 
propriately determine an improper image region such as 
a saturated region and a noise region, and an image 
sensing apparatus for outputting an image signal with 
the region corrected. 

It is another concern of the present invention to pro- 
vide an image processing method and apparatus for ap- 
propriately determining a saturated region, and an im- 
age sensing apparatus for outputting an image signal 
with the region corrected. 

An embodiment of the present invention provides 
an image processing method and apparatus for appro- 
priately determining a noise region, and an image sens- 
ing apparatus for outputting an image signal with the re- 
gion corrected. 

According to the present invention, there is provid- 
ed a method of determining the a threshold value setting 
algorithm on the basis of an average value, and a meth- 
od of determining the threshold value setting algorithm 
on the basis of the degree of change. 
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It is still another concern of the present invention to 
provide an image processing method and apparatus, 
which can set a threshold value to accurately determine 
an improper region, and an image sensing apparatus 
5 for outputting an image signal with the region corrected. 

According to an aspect of the present invention, a 
threshold value is determined on the basis of a standard 
image signal obtained in a standard exposure and a 
non-standard image signal obtained in a non-standard 
10 exposure. 

Additionally, according to the present invention, a 
threshold value may be applied to a standard image sig- 
nal to determine whether the signal is an improper im- 
age signal. 

15 Another embodiment of the present invention pro- 
vides an image processing method and apparatus, 
which can replace an image signal of a region deter- 
mined as an improper image signal with a proper image 
signal, and an image sensing apparatus for outputting 
an image signal with the region corrected. 

Another embodiment of the present invention pro- 
vides an image processing method and apparatus, 
which replace an image signal of a region of standard 
image determined as an improper image signal with a 
non-standard image signal obtained in a non-standard 
exposure to expand the dynamic range of the entire im- 
age signal, and an image sensing apparatus for output- 
ting an image signal with the region corrected. 

A still further concern of the present invention to pro- 
vide an image processing method and apparatus free 
from a hue offset, and an image sensing apparatus for 
outputting an image signal with the region corrected. 

According to another aspect of the present inven- 
tion, there is provided an image processing method and 
apparatus, which determine a saturated region by using 
the minimum value of threshold values set in units of 
colors, and an image sensing apparatus for outputting 
an image signal with the region corrected. 

According to the present invention, there may be 
provided an image processing method and apparatus, 
which determine a noise region by using the maximum 
value of threshold values set in units of colors, and an 
image sensing apparatus for outputting an image signal 
with the region corrected. 

It is still another concern of the present invention to 
provide an image processing method and apparatus 
free from a pseudo contour even when the dynamic 
range is expanded, and an image sensing apparatus for 
outputting an image signal with the region corrected. 

Other features and advantages of the present in- 
vention will be apparent from the following description 
taken in conjunction with the accompanying drawings, 
in which like reference characters designate the same 
or similar parts throughout the figures thereof. 

BRIEF DESCRIPTION OF THE DRAWINGS 

Figs. 1 A to 1 C are graphs for explaining a conven- 
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tional technique for expanding the dynamic range 
by correcting a saturated region; 
Figs. 2A to 2C are graphs for explaining another 
conventional technique for expanding the dynamic 
range by correcting a noise region; s 
Fig. 3 is a block diagram schematically showing the 
arrangement of an image sensing apparatus 
according to the first and second embodiments of 
the present invention; 

Fig. 4 is a view showing the arrangement of color 10 
filters of an image sensing device used in the image 
sensing apparatus in Fig. 3; 

Fig. 5 is a flow chart showing dynamic range 
expanding processing according to the first embod- 
iment; 15 
Fig. 6 is a view for explaining a mode of storing a 
standard image or a non-standard image in the 
apparatus shown in Fig. 3; 

Fig. 7 is a graph for explaining a method of setting 

a threshold value of the first and second embodi- 20 

ments; 

Fig. 8 is a view showing the standard image (proper 
exposure); 

Fig. 9 is a view showing the non-standard image 
(under exposure); 25 
Fig. 1 0 is a view showing a divided state of an image 
proper region and an improper region in the first 
embodiment; 

Fig. 1 1 is a graph for explaining another technique 
for setting a threshold value; and 30 
Fig. 12 is a view for explaining a technique for 
expanding the dynamic range according to the sec- 
ond embodiment. 



DETAILED DESCRIPTION OF THE PREFERRED 35 
EMBODIMENTS 

The present invention will be described below in de- 
tail with reference to embodiment to which the present 
invention is applied. In these embodiments, an image 40 
processing apparatus is applied to an image sensing ap- 
paratus. 

<Arrangement of Apparatus> 

45 

Fig. 3 is a block diagram of an image sensing ap- 
paratus according to an embodiment of the present in- 
vention. 

Referring to Fig. 3, reference numeral 1 denotes a 
lens on which image light from an object is incident; 2, so 
an optical low-pass filter; and 3, an image sensing de- 
vice such as a CCD. Complementary color filters of four 
colors arranged as shown in Fig. 4 are bonded to the 
light-receiving surface of the image sensing device 3. 

Referring to Fig. 3, reference numeral 4 denotes an 55 
A/D converter for converting an image signal from the 
image sensing device 3 into a digital signal; 5, a color 
separation circuit for separating image data from the 
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A/D converter 4 into four color image data (magenta Ma, 
yellow Ye, cyan Cy, and green Gr); 6, a switch; 7, a mem- 
ory for storing four color image data; 8 : a low-pass filter; 
9, a dynamic range expander; 10, a signal processing 
unit; 11, a system controller; and 12, an interpolation 
processing circuit. 

The operation of each circuit shown in Fig. 3 will be 
described below. 

An object image (not shown) is projected onto the 
image sensing device 3 through the lens 1 and the op- 
tical low-pass filter 2. An image signal output from the 
image sensing device 3 is converted into a digital signal 
by the A/D converter 4 and separated into four chromi- 
nance signals of Ma (magenta), Gr (green) : Cy (cyan) 
and Ye (yellow) by the color separation circuit 5. The 
system controller 11 controls the operations of all the 
circuits and processing units. In the image sensing ap- 
paratus of this embodiment, two modes, i.e., an "ex- 
panding mode" for expanding the dynamic range and a 
"normal mode" for not expanding the dynamic range are 
set. The controller 11 determines on the basis of an out- 
put value from the image sensing device 3 or a request 
from the user whether the dynamic range need be ex- 
panded. In the "normal mode", the switch 6 is connected 
to the signal processing unit 1 0 side, and an output from 
the color separation circuit 5 is immediately sent to the 
signal processing unit 1 0. The signal processing unit 10 
performs a known matrix calculation of the image data 
and outputs the image data as a luminance signal and 
a color difference signal. 

A case wherein the "expanding mode" for expand- 
ing the dynamic range is set will be described below. We 
propose two techniques of dynamic range expanding 
processing in the expanding mode. In the first technique 
(first embodiment), an image region detected as a sat- 
urated region because of the narrow dynamic range of 
the image sensing device is replaced with data of an 
image obtained not to generate a saturated region. In 
the second technique (second embodiment), in addition 
to processing according to the first technique, a dark- 
noisy region is replaced with noise-free image data. 

<First Embodiment 

In the expanding mode, the switch 6 is connected 
to the memory 7 side. An output from the color separa- 
tion circuit 5 is guided to the memory 7 through the 
switch 6 and stored in the memory 7 in units of chromi- 
nance signals. The interpolation processing circuit 12 
checks the chrominance signals stored in the memory 
7, interpolates omitted pixels for each color, and stores 
the signals in the memory 7 again. Highfrequency com- 
ponents are removed from the four chrominance signals 
after pixel interpolation processing, which are stored in 
the memory 7. Thereafter the chrominance signals are 
sent to the dynamic range (to be referred to as a D-range 
hereinafter) expander 9. 

There are two input paths from the memory 7 to the 
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expander 9. A path 9a is an input path from the filter 8, 
as described above. A path 9b is a direct input path from 
the memory 7. More specifically, image data received 
by the D-range expander 9 through the path 9b is not 
processed through the low-pass filter. On the other 
hand, image data received by the D-range expander 9 
through the path 9a is processed through the low-pass 
filter, as described above. When the D-range of an im- 
age is to be expanded by the D-range expander 9, a 
threshold value is determined on the basis of image data 
processed through the low-pass filter (i.e., image data 
accessed through the path 9a). Thereafter, this thresh- 
old value is used to perform D-range expanding 
processing of image data accessed through the path 9b. 
The reason why the threshold value is determined using 
the image data processed through the low-pass filter is 
that it can be accurately determined by removing high- 
frequency components whether image data is saturated 
or noisy. 

Fig. 5 is a flow chart showing the control sequence 
of the controller 1 1 . In step S1 , the interpolation process- 
ing circuit 1 2 is caused to perform interpolation process- 
ing of four color image data stored in the memory 7 and 
return the interpolated image data to the memory 7. In 
step S2, the controller 11 causes the filter 8 to perform 
low-pass filter processing of the four color image data 
(interpolated) stored in the memory 7 and sends the fil- 
ter-processed image data to the D-range expander 9. 
Subsequently, the controller 11 causes the D-range ex- 
pander 9 to perform threshold value setting processing. 
More specifically, as described above, threshold values 
(Tm, Tg, Tc, and Ty) used in D-range expanding 
processing are determined on the basis of the four color 
image data after low-pass filter processing. The control- 
ler 11 stores the determined threshold values in a pre- 
determined memory (not shown). These threshold val- 
ues are used to determine a saturated region in D-range 
expanding processing. The method of determining the 
threshold values will be described later in detail. 

In step S5, the controller 11 selects the path 9b, and 
causes the processing circuit 9 to perform expand- 
ing/combining processing. In D-range expanding 
processing, the threshold values (Tm, Tg, Tc, and Ty) 
set for the respective four color image data in units of 
colors are used to determine a saturated region, and a 
region determined as a saturated region is replaced with 
image data of a non-standard image. Non-standard im- 
age is an image obtained under improper exposure, 
while standard image is one obtained under proper ex- 
posure. The D-range expanding processing will be de- 
scribed later in detail. The image data with an expanded 
D-range is returned to the memory 7 by the processing 
circuit 9. 

In step S6, the controller 11 causes the processing 
circuit 9 to perform smoothing processing. More specif- 
ically, the processing circuit 9 performs smoothing 
processing of the image data with the expanded D- 
range. The smoothing processing is performed to re- 
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move a pseudo contour which may be generated in ac- 
cordance with D-range expanding processing. 

In this manner, the four chrominance signals with 
the expanded D-range are output from the signal 
s processing unit 10 as luminance and color difference 
signals (step S9). 

The operation of the D-range expander 9 according 
to the first embodiment will be described below in detail. 

In the expanding processing of the first embodi- 
ment, a saturated region present in an image 1000 (Fig. 
6) obtained upon sensing an object in a proper exposure 
(to be referred to as a standard image hereinafter) is 
replaced with a corresponding region of an image 1002 
obtained upon sensing the object in an improper expo- 
sure (under exposure) (to be referred to as a non-stand- 
ard image hereinafter). Six regions shown in Fig. 6 are 
provided to each of the four colors. 

Prior to threshold value setting, the luminance level 
of the standard image must coincide with that of the non- 
standard image. As a technique for adjusting the lumi- 
nance level, a technique proposed by the present appli- 
cant in Japanese Patent Application No. 5-271940, 
5-271941, 5-271942, 5-271943, 5-272117, or 5-281008 
(i.e., a technique described with reference to Figs. 1A 
to 1 C or 2A to 2C) is used. Therefore, a circuit and pro- 
gram for realizing this technique is implemented in the 
processing circuit 9. 

After the luminance level of the standard image 
1000 sensed in a proper exposure is adjusted in corre- 
spondence with that of the non-standard image 1002 
sensed in an improper exposure, the processing circuit 
9 performs processing of determining the thresholds 
values (Tm, Tg, Tc, and Ty) for determining whether a 
saturated region is present. This threshold value deter- 
mining processing is performed as follows. As shown in 
Fig. 7, the pixel value of the image signal of the standard 
image (or the image signal of the non-standard image) 
after luminance level adjustment (in Fig. 7, the signal of 
the non-standard image) is plotted along the abscissa, 
and the average value of differences between the pixel 
values of pixels of the standard image 1000 and those 
of corresponding pixels of a non-standard image 1001 , 
i.e., the average value of limproper image - proper im- 
agel is plotted along the ordinate. A value (T in Fig. 7) 
on the abscissa at a position slightly before the value of 
limproper image - proper imagel increases is set as a 
threshold value for detecting a saturated region. 

The meaning of the technique for determining the 
threshold value T in Fig. 7 will be described below with 
reference to Figs. 8 to 10. 

Fig. 8 is a view showing the distribution of pixel val- 
ues in part of the standard image of an object (image 
obtained in proper exposure). The characteristic of this 
standard image roughly lies in that the lower right portion 
is bright (pixel values are large) : and the upper left por- 
tion is dark (pixel values are small). 

Fig. 9 is a view showing the pixel value distribution 
of a region of the non-standard image obtained upon 
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sensing the object in under exposure, which region cor- 
responds to the region in Fig. 8. The non-standard im- 
age shown in Fig. 9 has been processed such that the 
luminance level coincides with that of the standard im- 
age in Fig. 8. More specifically, the pixel value of the 
non-standard image is multiplied by the luminance level 
adjustment value K. Fig. 10 is a view showing the abso- 
lute values of differences between the pixel values of 
pixels of the standard image in Fig. 8 and those of cor- 
responding pixels of the non-standard image in Fig. 9, 
i.e., the values of limproper image x K - proper imagel. 

Pixels in a region of the standard image of an object, 
which region is saturated due to the narrow D-range of 
the image sensing device, have a significant difference 
in pixel value from those in the corresponding region of 
the non-standard image obtained upon sensing the re- 
gion in under exposure. Therefore, the value of limprop- 
er image x K - proper imagel in such a region is consid- 
erably far from zero. Such a region appears as a region 
IX in Fig. 9 or 10. Pixels in a region of the standard im- 
age, which region is neither saturated nor nearly satu- 
rated, have no difference from those in the correspond- 
ing region of the non-standard image sensed in under 
exposure. Therefore, the value of limproper image x K 
- proper imagel in such a region is zero or close to zero. 
Such a region appears as a region VIII in Fig. 9 or 10. 
In Fig. 9 or 10, a region VII is a dark-noisy region of the 
non-standard image. 

In this manner, when a threshold value Tx for deter- 
mining a saturated region for one color is determined, 
threshold values for the remaining colors are also de- 
termined accordingly. 

Modification of First Embodiment 

Another technique for setting a threshold value for 
determining a saturated region will be described with 
reference to Fig. 11. 

In this technique, as shown in Fig. 11, pixel values 
of a standard image are plotted along the X-axis, and 
pixel values of a non-standard image are plotted along 
the Y-axis. The pixel value of a pixel of the standard im- 
age is plotted along the X-axis, and that of a correspond- 
ing pixel of the non-standard image is plotted along the 
Y-axis. As for a curve 2000 obtained by this plotting, a 
proper region which is neither saturated nor dark noise 
has a gradient of about one while a saturated region has 
a gradient of one or more. Therefore, a threshold value 
is set at the position represented by T in Fig. 11. 

Replacement of Saturated Pixel Value> 

In the above manner, threshold value setting 
processing is performed by the D-range expander 9, 
and the threshold values (Tm, Tg, Tc, and Ty) are set 
(step S4 in Fig. 5). After this setting processing, by using 
the threshold values (Tm, Tg, Tc, and Ty) for the respec- 
tive colors, it is determined whether the image signals 
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of the four standard images of Ma, Gr, Ye, and Cy from 
the interpolation processing circuit 12 have a saturated 
region. If a saturated region is determined, the pixel val- 
ue of the region is replaced with the image signal of the 
5 non-standard image (non-standard image obtained in 
an under exposure). This replacement means that the 
D-range of the image sensing device is expanded by 
combining the standard image with the non-standard 
image. 

to The meaning of smoothing processing in step S6 
(Fig. 5) becomes obvious. More specifically, in this em- 
bodiment, a threshold value for determining a saturated 
region is set in units of colors. If one threshold value is 
set for all colors, the boundary region between the 

*5 standard image and the non-standard image in the com- 
bined image for each color is recognized as a pseudo 
contour because of the lack of dispersion in units of 
colors. As in this embodiment, when threshold values 
dedicated for the respective colors are set, the boundary 

20 region between the standard image and the non-stand- 
ard image changes in units of colors. As a result dis- 
persion occurs, and dispersion itself means a smoothing 
effect. Therefore, smoothing processing in step S6 can 
be omitted. 

25 

<Modification of First Embodiment 

In D-range expanding processing of the first em- 
bodiment, a saturated region of a standard image 

30 sensed in a proper exposure is replaced with the corre- 
sponding pixel value of a non-standard image sensed 
in an under exposure, thereby expanding the D-range. 
The present invention is not limited to this and can also 
be applied to a case wherein a noise image region of a 

3S standard image is replaced with corresponding image 
data of a non-standard image sensed in an improper ex- 
posure (over exposure) to expand the D-range. In this 
case, a graph as shown in Fig. 7 is prepared, and a value 
immediately before the value of Istandard image - non- 

40 standard image x Kl decreases from a non-zero value 
to reach zero is set as a threshold value. 

In addition, the following modification can also be 
proposed. In the first embodiment, a saturated region is 
replaced with a non-standard image obtained in an un- 

45 der exposure, or a noise region is replaced with a non- 
standard image obtained in an over exposure. In other 
words, two images (a standard image and a non-stand- 
ard image obtained in an under exposure : or a standard 
image and a non-standard image obtained in an over 

50 exposure) are combined with each other, thereby ex- 
panding the D-range of the image sensing device. The 
present invention is not limited to this and can also be 
applied to a case wherein three images, i.e. , a standard 
image, a non-standard image (under exposure), and an- 

55 other non-standard image (over exposure) are com- 
bined to expand the D-range. That is, two or more im- 
ages can be used for combination. 

According to the image processing method and ap- 
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paratus of the first embodiment, threshold values are 
calculated in correspondence with the color filters of the 
image sensing device, and threshold values corre- 
sponding to chrominance signals output from the image 
sensing device and color-separated are used to deter- 
mine regions. With this operation, erroneous determina- 
tion in noisy or saturated region determination, which is 
caused by the difference in color spectral sensitivity, can 
be prevented. Since a threshold values is set in units of 
colors, a pseudo contour in a division boundary region, 
which is generated in use of a specific threshold value, 
can be minimized. As a result, an image without satu- 
rated or noisy region and with an expanded D-range can 
be combined. 

<Second Embodiment 

In the first embodiment, to detect a saturated re- 
gion, a plurality of threshold values corresponding to the 
respective colors are used. This technique poses the fol- 
lowing problem of hue offset. Generally, when image 
signals of the respective colors are to be obtained by 
using different color filters of an imaging sensing device, 
chrominance signals constituting a pixel receive differ- 
ent exposure amounts. As a result, when region deter- 
mination is performed using different threshold values, 
the hue changes. For example, when color filters shown 
in Fig. 4 are used, image sensing devices having a sen- 
sitivity for Ye in the four colors are saturated at a rela- 
tively low illuminance as compared to the remaining 
colors, so the threshold value for Ye tends to become 
smaller than those of the remaining colors, though this 
depends on color temperature. For this reason, in image 
combining, an image signal from a standard image is 
used as a combined image of Ma, Gr, Cy which are not 
saturated while an image signal from a non-standard im- 
age is used as a combined image of saturated Ye. As a 
result, the hue offsets. 

In the second embodiment to be described below, 
the minimum value of the four threshold values corre- 
sponding to the respective colors is used as a threshold 
value to prevent the above-described hue offset. More 
specifically, in the above example, the threshold value 
for Ye is used for all the remaining colors. 

Since the processing technique of a D-range ex- 
pander 9 is different from that in the first embodiment, 
the arrangement of the second embodiment is the same 
as that of the first embodiment (arrangement in Fig. 3). 
Therefore, the arrangement of the image sensing appa- 
ratus of the second embodiment will be described with 
reference to Fig. 3, and the control sequence of a con- 
troller 11 will be described with reference to Fig. 12. 

Steps S1 1 and S1 2 in Fig. 1 2 are the same as steps 
S1 and S2 of the first embodiment, and a detailed de- 
scription thereof will be omitted. 

In step S13 of the second embodiment, threshold 
value setting processing and threshold value selecting 
processing are performed, and it is determined whether 
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the image signal is saturated, proper, or noisy. More 
specifically, threshold values Tms (for determining sat- 
uration) and Tmn (for determining a noise region) for 
magenta, threshold values Tgs (for determining satura- 

5 tion) and Tgn (for determining a noise region) for green, 
threshold values Tcs (for determining saturation) and 
Ten (for determining a noise region) for cyan, and 
threshold values Tys (for determining saturation) and 
Tyn (for determining a noise region) for yellow are set 

10 first. The threshold value setting technique is the same 
as that of the first embodiment (technique in Fig. 7 or 
11). Subsequently, the following calculation is per- 
formed to obtain a threshold value Ts for determining a 
saturated region: 

75 Ts = Min(Tms, Tgs : Tcs, Tys) 

In addition, the following calculation is performed to ob- 
tain a threshold value Tn for determining a noise region: 
Tn = Max(Tmn, Tgn, Ten, Tyn) 
The reason why the maximum value of Tmn, Tgn, 

20 Ten, and Tyn is used as the threshold value Tn for de- 
termining a noise region is as follows. An image sensing 
device having a large noise (i.e., a low sensitivity) gen- 
erates a dark-noisy region even at a high illuminance. 
Therefore, selecting the maximum value of the thresh- 

25 old values for the respective colors assures to discrim- 
inate dark-noisy regions for all the colors. Further, the 
two selected threshold values Ts and Tn are used for 
image data of four color standard images, thereby de- 
termining saturated regions, dark noise regions, and 

30 proper regions. In step S14, the standard image is di- 
vided in accordance with determination. 

In step S1 5, a region determined as a saturated re- 
gion is replaced with the image signal of a non-standard 
image obtained in an under exposure, and a region de- 

35 termined as a dark-noisy region is replaced with the im- 
age signal of a non-standard image obtained in an over 
exposure. 

In step S16, smoothing processing of the combined 
image is performed to eliminate the pseudo contour. In 
40 the first embodiment, the smoothing processing is op- 
tional. In the second embodiment, the smoothing 
processing is important because the frequency of pseu- 
do contour generation is increased by using the same 
threshold value. 

45 

<Modification of Second Embodiment 

In D-range expanding processing of the second em- 
bodiment, a saturated region of a standard image 

50 sensed in a proper exposure is replaced with the corre- 
sponding region of a non-standard image sensed in an 
improper exposure (under exposure); and a noise re- 
gion of the standard image is replaced with the corre- 
sponding region of the non-standard image sensed in 

55 an improper exposure (over exposure). The present in- 
vention is not limited to this and can also be modified 
such that only replacement of the saturated region of 
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the standard image with the corresponding region of the 2. 
non-standard image sensed in an under exposure, or 
only replacement of the noise region of the standard im- 
age with the corresponding region of the non-standard 
image sensed in an over exposure is performed. In this s 
case : two images, i.e., the standard image and the non- 
standard image (under or over exposure) suffices. 

As described above, according to the image 
processing method and apparatus of the second em- 
bodiment, of a plurality of image data obtained upon 10 
sensing an identical scene in different exposure 
amounts, a noise region of the standard image sensed 
in a proper exposure is replaced with the corresponding 
region of the non-standard image sensed in an over ex- 
posure, and a saturated region of the standard image is 
sensed in a proper exposure is replaced with the corre- 
sponding region of the non-standard image sensed in 
an under exposure, thereby combining an image with 
an expanded D-range. 

In the method of determining a dark noise or satu- 20 
rated region, threshold values are calculated in corre- 
spondence with the color fitters of the image sensing de- 
vice. When a saturated region is to be determined, the 
minimum value of the plurality of threshold values is 3. 
used, and when a dark noise region is to be determined, 25 
the maximum value of the plurality of threshold values 
is used. In use of the plurality of threshold values, the 
hue change caused when the chrominance signals con- 
stituting one pixel are generated from image signals ob- 
tained in different exposure amounts can be prevented. 30 4. 
As a result, an image without any dark noise or saturated 
region and with an expanded D-range can be obtained. 

As many apparently widely different embodiments 
of the present invention can be made without departing 
from the spirit and scope thereof, it is to be understood 35 
that the invention is not limited to the specific embodi- 
ments thereof except as defined in the appended 
claims. 5, 

40 

Claims 

1. An image processing method of determining 
whether an image signal output from an image 
sensing device through a predetermined color filter 45 
is improper, characterized by comprising: 6. 

the threshold value setting step (S4, S13) of 
setting a threshold value in correspondence 
with a color filter of said image sensing device; so 
and 

the determination step (S5, S1 3) of determin- 
ing, in units of . colors, whether each chromi- 
nance signal output from said image sensing 7. 
device and color-separated is improper by ss 
using the threshold value corresponding to 
each color. 



An image processing method of determining 
whether an image signal output from an image 
sensing device through a predetermined color filter 
is improper, characterized by comprising: 

the threshold value setting step (S4, S13) of 
setting a threshold value in correspondence 
with each filter of said image sensing device; 
the sensing step of sensing an image in 
advance by said image sensing device in a 
standard exposure and in a non-standard expo- 
sure to obtain a standard image signal and a 
non-standard image signal corresponding to 
each color; 

the determination step (S5, S1 3) of determining 
whether the standard image signal of one color 
is improper by using the threshold value set in 
correspondence with the color; and 
the replacement step (S5, S1 5) of replacing the 
standard image signal determined as an 
improper image signal with the corresponding 
non-standard image signal. 

The method according to claim 2, wherein the 
threshold value setting step includes the step of 
determining the threshold value corresponding to 
the color on the basis of the standard image signal 
and the non-standard image signal. 

The method according to claim 2, wherein the 
threshold value setting step includes the step of 
determining the threshold value corresponding to 
one color depending on whether a degree of 
change in non-standard image signal with respect 
to the standard image signal obtained in the sensing 
step is large. 

The method according to claim 2, wherein the 
threshold value setting step includes the step of 
obtaining the non-standard image signal in an expo- 
sure amount smaller than a standard exposure 
amount when it is to be determined whether the 
image signal obtained from said image sensing 
device is saturated. 

The method according to claim 2, wherein the 
threshold value setting step includes the step of 
obtaining the non-standard image signal in an expo- 
sure amount larger than a standard exposure 
amount when it is to be determined whether the 
image signal obtained from said image sensing 
device is dark-noisy. 

An image processing method of determining a sat- 
urated region of an image signal output from an 
image sensing device through a predetermined 
color filter, characterized by comprising: 



BNSDOCID: <EP 0713342A2_I_> 



8 



1 



15 

the threshold value setting step (S1 3) of setting 
a threshold value in units of colors in corre- 
spondence with a color filter of said image 
sensing device: and 

the determination step (S1 3) of determining the 
saturated region in each chrominance signal 
output from said image sensing device and 
color-separated by using a minimum threshold 
value. 

8. The method according to claim 7, wherein the 
threshold value setting step includes the steps of 

sensing an image in advance by said image 
sensing device in a standard exposure and in 
a non-standard exposure to obtain a standard 
image signal and a non-standard image signal 
corresponding to each color; and 
setting the threshold value for the color on the 
basis of the obtained standard image signal 
and the non-standard image signal of a certain 
color. 

9. The method according to claim 7, wherein the 
threshold value setting step comprises: 

the step of sensing an image in advance by said 
image sensing device in a standard exposure 
to obtain a standard image signal of one color; 
the step of sensing the image in advance by 
said image sensing device in a non-standard 
exposure to obtain a non-standard image sig- 
nal of the color; and 

the step of determining the threshold value cor- 
responding to the color on the basis of the 
obtained standard image signal and the non- 
standard image signal. 

10. The method according to claim 9, wherein the 
threshold value setting step includes the step of cal- 
culating the threshold value corresponding to one 
color by using an average value of a difference 
between the standard image signal and the non- 
standard image signal. 

11. The method according to claim 9, wherein the 
threshold value setting step includes the step of 
determining the threshold value corresponding to 
one color depending on whether a degree of 
change in non-standard image signal with respect 
to the standard image signal is large. 

1 2. An image processing method of determining a noise 
region of an image signal output from an image 
sensing device through a predetermined color filter, 
characterized by comprising: 

the threshold value setting step (S1 3) of setting 
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a threshold value in units of colors in corre- 
spondence with a color filter of said image 
sensing device; and 

the determination step (S1 3) of determining the 
5 noise region in each chrominance signal output 

from said image sensing device and color-sep- 
arated by using a maximum threshold value. 

13. The method according to claim 12, wherein the 
10 threshold value setting step includes the steps of 

sensing an image in advance by said image 
sensing device in a standard exposure and in 
a non-standard exposure to obtain a standard 
1 5 image signal and a non-standard image signal 

corresponding to each color, and 
setting the threshold value for the color on the 
basis of the obtained standard image signal 
and the non-standard image signal of a certain 
20 color. 

14. The method according to any one of claims 1 , 2 or 
12, wherein the determination step includes the 
step of determining the image signal as improper 

2S when the image signal is an image signal from a 
saturated image sensing device. 

15. The method according to any one of claims 1 , 2 or 

12, wherein the determination step includes the 
30 step of determining the image signal as improper 

when the image signal is an image signal from a 
low-sensitivity image sensing device. 

16. The method according to any one of claims 1 , 2 or 
35 12, wherein the determination step includes the 

step of applying the threshold value set in the 
threshold value setting step to a signal of a standard 
image sensed by said image sensing device in a 
standard exposure. 

40 

17. The method according to any one of claims 1 , 2 or 

13, wherein the threshold value setting step com- 
prises: 

45 the step of sensing an image in advance by said 

image sensing device in a standard exposure 
to obtain a standard image signal of one color; 
the step of sensing an image in advance by said 
image sensing device in a non-standard expo- 
50 sure to obtain a non-standard image signal of 

the color; and 

the step of determining the threshold value cor- 
responding to the color on the basis of the 
obtained standard image signal and the non- 
55 standard image signal. 

18. The method according to claim 17, wherein the 
threshold value setting step includes the step of caf- 
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culating the threshold value corresponding to one 
color by using an average value of a difference 
between the standard image signal and the non- 
standard image signal. 

5 

19. The method according to claim 17, wherein the 
threshold value setting step includes the step of 
determining the threshold value corresponding to 
one color depending on whether a degree of 
change in non-standard image signal with respect 10 
to the standard image signal is large. 

20. The method according to claim 17, wherein the 
threshold value setting step includes the step of 
obtaining the non-standard image signal in an expo- is 
sure amount smaller than a standard exposure 
amount when it is to be determined whether the 
image signal obtained from said image sensing 
device is saturated. 

20 

21. The method according to claim 17, wherein the 
threshold value setting step includes the step of 
obtaining the non-standard image signal in an expo- 
sure amount larger than a standard exposure 
amount when it is to be determined whether the zs 
image signal obtained from said image sensing 
device is dark-noisy. 

22. The method according to claim 7, further compris- 
ing the steps of obtaining a non-standard signal in 30 
a non-standard exposure, and replacing the image 
signal determined as the saturated region in the 
determination step with the non-standard image 
signal. 

35 

23. The method according to claim 1 2, further compris- 
ing the steps of obtaining a non-standard signal in 
a non-standard exposure, and replacing the image 
signal determined as the saturated region in the 
determination step with the non-standard image 40 
signal. 

24. The method according to claim 22, further compris- 
ing the step of performing smoothing processing of 

the replaced image. 45 

25. The method according to claim 23, further compris- 
ing the step of performing smoothing processing of 
the replaced image. 

so 

26. An image processing apparatus which determines 
whether an image signal output from an image 
sensing device through a predetermined color filter 
is improper, characterized by comprising: 

ss 

threshold value setting means (S4, S1 3) for set- 
ting a threshold value in correspondence with 
a color filter of said image sensing device; and 



determination means (S5, S13) for determin- 
ing, in units of colors, whether each chromi- 
nance signal output from said image sensing 
device and color-separated is improper by 
using the threshold value corresponding to 
each color. 

27. An image processing apparatus which determines 
whether an image signal output from an image 
sensing device through a predetermined color filter 
is improper, characterized by comprising: 

threshold value setting means (S4, S1 3) for set- 
ting a threshold value in correspondence with 
a color filter of said image sensing device; 
sensing means for sensing an image in 
advance by said image sensing device in a 
standard exposure and in a non-standard expo- 
sure to obtain a standard image signal and a 
non-standard image signal corresponding to 
each color; 

determination means (S5, S1 3) for determining 
whether the standard image signal of one color 
is improper by using the threshold value set in 
correspondence with the color; and 
replacement means (S5, S15) for replacing the 
standard image signal determined as an 
improper image signal with the corresponding 
non-standard image signal. 

28. An image processing apparatus lor determining a 
saturated region of an image signal output from an 
image sensing device through a predetermined 
color filter, characterized by comprising: 

threshold value setting means (S13) for setting 
a threshold value in units of colors in corre- 
spondence with a color filter of said image 
sensing device; and 

determination means (S1 3) for determining the 
saturated region in each chrominance signal 
output from said image sensing device and 
color-separated by using a minimum threshold 
value. 

29. An image processing apparatus for determining a 
noise region of an image signal output from an 
image sensing device through a predetermined 
color filter, characterized by comprising: 

threshold value setting means (S13) for setting 
a threshold value in units of colors in corre- 
spondence with a color filter of said image 
sensing device; and 

determination means (S1 3) for determining the 
noise region in each chrominance signal output 
from said image sensing device and color-sep- 
arated by using a maximum threshold value. 
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